Skip to content

fix: MultiStart gradient step loop crashes on a real iterations_per_full_update cadence - #1421

Merged
Jammy2211 merged 3 commits into
mainfrom
feature/multistart-cadence-int-cast
Jul 27, 2026
Merged

fix: MultiStart gradient step loop crashes on a real iterations_per_full_update cadence#1421
Jammy2211 merged 3 commits into
mainfrom
feature/multistart-cadence-int-cast

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Closes #1420.

Summary

The MultiStart gradient step loop does for _ in range(iterations), but
AbstractSearch.__init__ stores iterations_per_full_update as a float
(abstract_search.py:219) so the inf-like 1e99 config default is
representable. The crash was latent because min(1e99, steps_remaining) returns
the int operand — only a user-supplied cadence below the remaining budget
reaches range and raises:

TypeError: 'float' object cannot be interpreted as an integer

Six RAL chain jobs (331182–331190) died on this back-to-back during the wsdev#117
Pix-Prodigy CPU campaign, using iterations_per_full_update=50 with
n_steps=3000.

The cast is applied at the consumer, via a new
AbstractMultiStartGradient._steps_in_chunk, rather than in the shared float()
coercion — that coercion exists to accept 1e99 and is used by every other
search, so changing it there is the regression-prone option. The helper also
makes the defect testable without JAX: _fit requires jax, optax and a
JAX-traceable Analysis, and the library unit suite is NumPy-only.

Commits 2 and 3 come from review, and are the interesting part of this PR.

Commit 2 fixed a defect in commit 1: int truncates towards zero, so a
fractional cadence below 1 gives range(0) — no steps run, total_steps never
advances, and the enclosing while spins forever re-running perform_update.
Commit 1 alone would have traded a loud TypeError for a silent hang, which on
a cluster burns the whole allocation. It floored the chunk at 1.

Commit 3 replaced that floor, after an independent adversarial review (Codex
gpt-5.6-sol) made two points that hold up:

  • max(1, int(...)) silently launders invalid input-5, 0.5 and 50.9
    all became a plausible-looking cadence, so a typo would quietly run a schedule
    the user never asked for. That is the silent-guard pattern this codebase
    removes rather than adds. It now raises a ValueError naming the value.
  • The "chunk can never overshoot steps_remaining" claim held only because
    n_steps is an integer
    . The loop guard proves steps_remaining > 0, not
    >= 1, so a float n_steps=2.5 leaves a 0.5 remainder that truncates to a
    zero-length chunk and hangs. n_steps is annotated int but never validated,
    so it is validated too. The two checks together restore the invariant the
    floor was papering over, which is why the floor could be removed rather than
    kept alongside them.

API Changes

None. The only new symbol is the private
AbstractMultiStartGradient._steps_in_chunk; no public class, method, signature,
argument, default or config key changed. Behaviour on the default path is
bit-identical — with the packaged 1e99 cadence the expression already returned
the int steps_remaining, and int()/max(1, ...) leave that untouched.
Downstream workspaces and notebooks need no migration.

The change is purely corrective: inputs that previously crashed now work.
Nothing that previously ran behaves differently.

Testing

  • pytest test_autofit/ -x in the task worktree → 1541 passed, 1 skipped.
  • New NumPy-only tests in
    test_autofit/non_linear/search/mle/test_multi_start_gradient.py: a real
    cadence below the budget returns an int that range() consumes; the chunk
    clamps to steps_remaining; the 1e99 default still gives a single
    whole-budget chunk; a falsy cadence falls back to n_steps; an unusable
    cadence (0.5, 50.9, -5) and an unusable n_steps (2.5, 0, -10)
    each raise ValueError.
  • A wiring guard, also from the review: every other test drives
    _steps_in_chunk directly, so all of them would still pass if _fit went
    back to computing the chunk inline — the exact regression this PR is about.
    _fit can't be executed from this suite (jax + optax + a JAX-traceable
    Analysis; the library suite is NumPy-only), so the call site is asserted at
    the source level.
  • Regression pinned: removing the int() cast fails exactly one new test
    (test__steps_in_chunk__real_cadence_is_an_int_range_can_consume).
  • Reproduced on main first: range(min(50.0, 3000)) raises the reported
    TypeError.

Validation checklist

  • Testspytest test_autofit/ -x: 1541 passed, 1 skipped. No public API
    change, so downstream dependent suites are n/a.
  • Smoke — curated smoke_tests.txt across all six workspaces with the task
    worktree's activate.sh sourced: 50 passed, 7 failed, 3 skipped/missing.
    All 7 failures are pre-existing — re-running each against main under
    identical conditions reproduces exactly the same 7 (they are the
    jax_likelihood parity scripts, which the smoke profile runs with
    PYAUTO_DISABLE_JAX=1). A further 5 scripts failed only in the parallel
    sweep and pass on the branch when re-run sequentially, matching main
    contention over shared output//dataset/ state in the runner, not a
    regression.
  • Review — two reviewers. The Brain review faculty returned FINDINGS on
    pass 1 (the truncation-to-zero hang → commit 2) and CLEAN on pass 2. An
    independent adversarial review (Codex gpt-5.6-sol, xhigh) then produced
    commit 3, corrected the follow-up list below, and surfaced two pre-existing
    bugs now filed separately.
  • HeartYELLOW, score 65, 2026-07-27T12:11:07Z, no RED reasons.
    The reason set was acknowledged by the human at this launch, verbatim:
    workspace validation not passing (13 failed, 2026-07-21T19-05-22Z);
    33 stale parked script(s);
    manifest drift: tenant firewall (organ code) — 5 mismatch(es) vs PyAutoMind/repos.yaml;
    and the stale-tier release validation stale: source moved since rehearsal (PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens).
    None touches autofit/non_linear/search/mle/.

Follow-ups (deliberately not in this PR)

  1. Two sibling searches carry the same defect, both confirmed reproducible
    mcmc/emcee/search.py:206 (EnsembleSampler.sample does range(iterations)
    with no cast) and mcmc/blackjax/nuts/search.py:291 (jax.random.split(key, 50.0) raises the same TypeError). An independent adversarial review
    (Codex gpt-5.6-sol) checked every consumer empirically and corrected an
    earlier, wider claim of mine: mcmc/zeus/search.py:242 is safe (zeus casts
    internally via self.nsteps = int(iterations)), and mle/bfgs/search.py:171,
    nest/dynesty/.../abstract.py:365 and nest/nautilus/search.py:477 are
    tolerated (comparison limits / arithmetic only). See issue fix: MultiStart gradient step loop crashes on a real iterations_per_full_update cadence #1420 for the
    verified table.
  2. Two pre-existing MultiStart bugs, both confirmed by reading the call
    sites and filed as PyAutoMind drafts:
    • the final perform_update runs twice_fit emits one with
      during_analysis=False at search.py:432, then start_resume_fit emits
      the same at abstract_search.py:704. Every other search
      (emcee:238, bfgs:219, nautilus:438) passes during_analysis=True
      unconditionally inside _fit, so MultiStart is the outlier and doubles
      the cost of final output, latents, visualization and profiling.
    • a completed stop_reason="max_steps" search resumed with a larger
      n_steps keeps the stale stop reason in every intermediate checkpoint
      (search.py:413-416 only reassigns on convergence or the new ceiling).
  3. The workspace hotfix should be removed once this merges — the
    post-construction int overwrite in
    autolens_workspace_developer/searches_minimal/pix_prodigy.py (branch
    feature/pix-prodigy-cpu), which belongs to the live pix-prodigy-cpu task
    (wsdev#117).

🤖 Generated with Claude Code

Jammy2211 and others added 2 commits July 27, 2026 14:58
The step loop does `for _ in range(iterations)`, but
`AbstractSearch.__init__` stores `iterations_per_full_update` as a float
(abstract_search.py:219) so the inf-like 1e99 config default is
representable. The crash was latent because `min(1e99, steps_remaining)`
returns the int operand — only a user-supplied cadence *below* the
remaining budget reaches `range` and raises `TypeError: 'float' object
cannot be interpreted as an integer`. Six RAL chain jobs died on it.

Cast at the consumer via a new `_steps_in_chunk` helper rather than
changing the shared float coercion, which every other search relies on.
The helper also makes the defect testable without JAX: `_fit` needs jax,
optax and a JAX-traceable Analysis, and the library suite is NumPy-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review of the previous commit: `int` truncates towards zero, so a
fractional cadence below 1 (e.g. iterations_per_full_update=0.5) yields
`range(0)` — no steps run, `total_steps` never advances, and the
enclosing while-loop spins forever re-running `perform_update`. That
traded a loud TypeError for a silent hang, which on HPC burns the whole
allocation. Floor at 1: the slowest cadence that still progresses, and
it can never overshoot `steps_remaining` (>= 1 whenever the loop runs).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows an adversarial review of the previous commit. `max(1, int(...))`
silently laundered invalid input: -5, 0.5 and 50.9 all became a
plausible-looking cadence, so a typo would quietly run a schedule the
user never asked for. Validate instead, and name the bad value.

The same review showed the "chunk can never overshoot steps_remaining"
claim held only because `n_steps` is an integer — the loop guard proves
steps_remaining > 0, not >= 1, so a float n_steps=2.5 leaves a 0.5
remainder that truncates to a zero-length chunk and hangs. n_steps is
annotated `int` but never validated, so it is checked too; both checks
together restore the invariant the floor was papering over.

Also adds a wiring guard: every other test drives _steps_in_chunk
directly, so all of them would still pass if _fit went back to computing
the chunk inline — the exact regression this fix is about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit e217292 into main Jul 27, 2026
5 checks passed
@Jammy2211
Jammy2211 deleted the feature/multistart-cadence-int-cast branch July 27, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: MultiStart gradient step loop crashes on a real iterations_per_full_update cadence

1 participant